support multiple ranks and alphas for LoRA #873

pacman100 · 2023-08-28T10:34:53Z

What does this PR do?

Add support for different ranks and alphas for LoRA tuner.
Add support for multiple active adapters

To Do:

Tests
Documentation
Example

HuggingFaceDocBuilderDev · 2023-08-28T10:40:03Z

The documentation is not available anymore as the PR was closed or merged.

src/peft/tuners/lora.py

Co-Authored-By: Benjamin Bossan <[email protected]>

src/peft/tuners/lora.py

Co-Authored-By: Benjamin Bossan <[email protected]>

BenjaminBossan

LGTM, thanks for adding this so quickly.

There are a few things I'd like to see in relation to this PR, but given that it helps with the diffusers PR, I'm fine with adding those later to speed things up:

unit tests
an example in the docs or examples folder
similar config options for other tuners where it makes sense, like AdaLoRA
maybe: regex matching

kovalexal · 2023-09-08T16:48:47Z

Hi, @pacman100! This feature is a super useful addition!

I am currently working on proper support of LoRA-C3Lier for Stable Diffusion. It's a special type of LoRA for which the Conv2d 3x3 layers may have different ranks and alphas. So this code will be super-useful for me to finish implementing this type of LoRA.

Do you by chance know what is the ETA for this feature to get into main branch?

kovalexal · 2023-09-12T18:02:27Z

@pacman100, @BenjaminBossan sorry to disturb you, I've been trying out the code from this branch, and I've faced and issue which is related to the usage of LoRAs with provided rank_pattern and alpha_pattern.

In short - if you need to create 2 adapters for model - first one without provided rank_pattern and alpha_pattern, second one with rank_pattern and alpha_pattern provided. After adding second adapter you can see, that ranks (and also alphas) for custom layers are not taken into account.

Here is a demo notebook which demonstrates how to reproduce this issue.

As far as I see, there is a possible problem in this specific lines:

peft/src/peft/tuners/lora/model.py

Lines 164 to 166 in 95fa3c1

    
           current_key = optional_kwargs["current_key"] 
        
           r = lora_config.rank_pattern.get(current_key, lora_config.r) 
        
           alpha = lora_config.alpha_pattern.get(current_key, lora_config.lora_alpha)

When the first adapter is initialized - the name of the module is passed as usual (e.g. "path.to.module"), but when the second adapter is added - the name of the module is nested in peft prefix (e.g. "base_model.model.path.to.module"), so that's why custom alphas and ranks are not propagated properly.

kovalexal · 2023-09-14T09:13:58Z

@pacman100 @BenjaminBossan I've addressed this issue and worked on 3 out of 4 comments @BenjaminBossan left (all except AdaLoRA).

Please let me know if there a way for me to somehow add my modifications to this PR.

pacman100 · 2023-09-15T11:07:52Z

Hello @kovalexal, Thank you for actively working on this! Please raise a PR to merge your changes in this PR, we will be happy to quickly review it and incorporate the changes.

pacman100 · 2023-09-15T11:46:32Z

Do you by chance know what is the ETA for this feature to get into main branch?

Next week is the plan

…ort of LoRA-C3Lier conversion (#937) * Fixed multirank multialpha for sequential loras, added tests, fixed docs * Refactored kohya_ss conversion script for proper support of LoRA-C3Lier * Fixed styling * Removed old comment from docstring

…ll the child classes

Co-Authored-By: Benjamin Bossan <[email protected]>

src/peft/tuners/adalora/bnb.py

Co-Authored-By: Younes Belkada <[email protected]>

younesbelkada

Thanks a lot Sourab! I left few minor comments as a first pass ! Let me know what do you think

src/peft/tuners/adalora/gptq.py

src/peft/tuners/adalora/bnb.py

src/peft/tuners/adalora/layer.py

src/peft/tuners/lora/bnb.py

src/peft/tuners/lora/gptq.py

src/peft/tuners/lora/layer.py

BenjaminBossan

Great work, this is a huge addition to PEFT.

I left a couple of comments, please take a look if they make sense. I have mostly reviewed the lora code closely, as I think IA³ and AdaLora just follow the same logic, so some of my comments apply there as well.

examples/lora_dreambooth/convert_kohya_ss_sd_lora_to_peft.py

src/peft/tuners/lora/model.py

src/peft/tuners/lora/layer.py

Co-authored-by: Younes Belkada <[email protected]>

BenjaminBossan

Overall, this LGTM, thanks for addressing the issues. I think there is one bug, however, please check my comment.

It is important that we also add tests for multiple active adapters, but if this is time critical, I'm okay with adding them later. Docs/examples can also be added later.

src/peft/tuners/lora/model.py

BenjaminBossan

LGTM, thanks for addressing the comments.

Let's not forget to add tests + docs + examples later.

I think using multiple adapters with dropout could be problematic, as more and more values will be set to 0. But I guess that's on the user to address, not sure if there is anything that we could do about it.

pacman100 · 2023-09-22T11:43:27Z

I think using multiple adapters with dropout could be problematic, as more and more values will be set to 0. But I guess that's on the user to address, not sure if there is anything that we could do about it.

I believe the support for multiple active adapters should be used only during inference in eval mode. In that case, dropout won't be an issue.

BenjaminBossan · 2023-09-22T11:47:12Z

I believe the support for multiple active adapters should be used only during inference in eval mode. In that case, dropout won't be an issue.

I see. But in theory, it would also work during training, right? And there isn't anything in the code right now that would prevent that.

younesbelkada

Great work @pacman100 , as always! Thanks!

whybeyoung · 2023-10-14T02:02:04Z

Any clearly examples for this pr?

mcc311 · 2023-10-14T15:04:50Z

Could anyone provide any example here? Thanks!

BenjaminBossan · 2023-10-23T12:42:42Z

Sorry that we haven't added any documentation yet. The idea is that you can do something like this:

config = LoraConfig(
    r=8,
    rank_pattern={"foo": 16},
    lora_alpha=10,
    alpha_pattern={"bar": 20, "baz": 30},
)

In this case, all LoRA layers would use rank 8, except for layers that match "foo", which would have rank 16. Similarly, LoRA alpha would be 10 by default except for "bar", which would be 20, and "baz", which would be 30.

* support multiple ranks and alphas * Update lora.py * Update lora.py * commit suggestions Co-Authored-By: Benjamin Bossan <[email protected]> * address comments Co-Authored-By: Benjamin Bossan <[email protected]> * Fixed multirank + multialpha for sequential LoRAs, added correct support of LoRA-C3Lier conversion (huggingface#937) * Fixed multirank multialpha for sequential loras, added tests, fixed docs * Refactored kohya_ss conversion script for proper support of LoRA-C3Lier * Fixed styling * Removed old comment from docstring * shift `scale_layer`/`unscale_layer` to `LoraLayer` class to support all the child classes * support multiple active adapters * add `active_adapters` property Co-Authored-By: Benjamin Bossan <[email protected]> * fix bug related to active adapter of `ModulesToSaveWrapper` * revert the change wrt active_adapter assignment Co-Authored-By: Younes Belkada <[email protected]> * Apply suggestions from code review Co-authored-by: Younes Belkada <[email protected]> * Apply suggestions from code review Co-authored-by: Younes Belkada <[email protected]> * addressing comments * address comments * address comment --------- Co-authored-by: Benjamin Bossan <[email protected]> Co-authored-by: Alexander Kovalchuk <[email protected]> Co-authored-by: Younes Belkada <[email protected]>

support multiple ranks and alphas

248510f

BenjaminBossan reviewed Aug 28, 2023

View reviewed changes

src/peft/tuners/lora.py Outdated Show resolved Hide resolved

pacman100 added 2 commits August 28, 2023 17:09

Update lora.py

436c40e

Update lora.py

3253491

pacman100 marked this pull request as ready for review August 28, 2023 11:54

commit suggestions

873a091

Co-Authored-By: Benjamin Bossan <[email protected]>

BenjaminBossan reviewed Aug 28, 2023

View reviewed changes

src/peft/tuners/lora.py Outdated Show resolved Hide resolved

src/peft/tuners/lora.py Outdated Show resolved Hide resolved

pacman100 and others added 2 commits August 29, 2023 17:21

Merge branch 'main' into smangrul/lora-supoort-multirank-multialpha

c411d2b

address comments

95fa3c1

Co-Authored-By: Benjamin Bossan <[email protected]>

BenjaminBossan approved these changes Aug 29, 2023

View reviewed changes

kovalexal mentioned this pull request Sep 15, 2023

Fixed multirank + multialpha for sequential LoRAs, added correct support of LoRA-C3Lier conversion #937

Merged

kovalexal and others added 4 commits September 18, 2023 17:19

Merge branch 'main' into smangrul/lora-supoort-multirank-multialpha

272c363

shift scale_layer/unscale_layer to LoraLayer class to support a…

40728ea

…ll the child classes

support multiple active adapters

b3b3db6

BenjaminBossan mentioned this pull request Sep 21, 2023

[PEFT / LoRA] PEFT integration - text encoder huggingface/diffusers#5058

Merged

add active_adapters property

3cacc34

Co-Authored-By: Benjamin Bossan <[email protected]>

pacman100 requested review from BenjaminBossan and younesbelkada September 21, 2023 12:32

younesbelkada reviewed Sep 21, 2023

View reviewed changes

src/peft/tuners/adalora/bnb.py Outdated Show resolved Hide resolved

pacman100 and others added 2 commits September 21, 2023 18:19

fix bug related to active adapter of ModulesToSaveWrapper

9ca063f

revert the change wrt active_adapter assignment

4b62bc5

Co-Authored-By: Younes Belkada <[email protected]>

younesbelkada reviewed Sep 21, 2023

View reviewed changes

BenjaminBossan reviewed Sep 21, 2023

View reviewed changes

pacman100 and others added 4 commits September 21, 2023 19:16

Apply suggestions from code review

6917f1f

Co-authored-by: Younes Belkada <[email protected]>

Apply suggestions from code review

61c2e34

Co-authored-by: Younes Belkada <[email protected]>

Merge branch 'main' into smangrul/lora-supoort-multirank-multialpha

6ede6b0

addressing comments

bae800c

BenjaminBossan requested changes Sep 22, 2023

View reviewed changes

src/peft/tuners/lora/model.py Show resolved Hide resolved

src/peft/tuners/lora/model.py Outdated Show resolved Hide resolved

pacman100 added 2 commits September 22, 2023 14:36

address comments

385c327

address comment

425bf49

pacman100 requested review from BenjaminBossan and younesbelkada September 22, 2023 11:28

BenjaminBossan approved these changes Sep 22, 2023

View reviewed changes

younesbelkada approved these changes Sep 22, 2023

View reviewed changes

pacman100 merged commit 1c0654b into main Sep 22, 2023

BenjaminBossan mentioned this pull request Nov 7, 2023

DOC: List of TODOs for the documentation #1089

Closed

10 tasks

BenjaminBossan mentioned this pull request Jan 10, 2024

AdaLora rank_pattern is None, Docs and Warnings and Integrations should be improved up on. #1327

Closed

4 tasks

pacman100 deleted the smangrul/lora-supoort-multirank-multialpha branch February 20, 2024 05:48

support multiple ranks and alphas for LoRA #873

support multiple ranks and alphas for LoRA #873

Uh oh!

Conversation

pacman100 commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan left a comment • edited by pacman100 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kovalexal commented Sep 8, 2023

Uh oh!

kovalexal commented Sep 12, 2023

Uh oh!

kovalexal commented Sep 14, 2023

Uh oh!

pacman100 commented Sep 15, 2023

Uh oh!

pacman100 commented Sep 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

pacman100 commented Sep 22, 2023

Uh oh!

BenjaminBossan commented Sep 22, 2023

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

whybeyoung commented Oct 14, 2023

Uh oh!

mcc311 commented Oct 14, 2023

Uh oh!

BenjaminBossan commented Oct 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

pacman100 commented Aug 28, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Aug 28, 2023 •

edited

Loading

BenjaminBossan left a comment •

edited by pacman100

Loading

pacman100 commented Sep 15, 2023 •

edited

Loading